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Abstract 

Background: Asynchronous discussion boards provide opportunities for participants in online courses to engage with course content and 
extend learning through the process of discussion. However, simply requiring online discussion does not guarantee high-quality discussion posts 
and interactions. 

Goals: This study examined the extent to which discussion boards in the eRead Ohio online professional development courses support and 
extend learning and allow students to engage more deeply with the course materials. The study also examined the characteristics (i.e., rigor, 
relevance, etc.) of the discussion board prompts and the influence of the prompt on subsequent discussions. 

Research Methods: Researchers analyzed discussion boards from three of the 150 eRead Ohio online courses using a coding system developed 
by Puntambekar, which allows for coding of cognitive demand and the extent to which the posting is related to theory or course material aligned to 
the updated Bloom’s taxonomy. 

Results: Teacher-participants reported increased insight into their practice and many described intended changes in practices. However, the 
vast majority of discussion prompts did not insist that students reflect and comment on course content in their response; few participants referred to 
course content to support current practices or changes in practice. Participants answered instructors’ initial question and seldom challenged peers 
to higher levels of analysis or reflection. Although differences were evident among the level of prompts posed by instructors, neither the prompts 
nor responses reached high levels of the coding rubric, the level of the prompt was correlated with the level of responses that students produced. 
Questions that were rated at higher levels on the scoring rubric, in general, generated responses that fell at higher levels on the rubric. The methods 
applied for analysis in this study provide an example of a clear coding system to analyze discussion board activity during or after the completion of 
a course. 


Key Words: Online teacher professional development (OTPD), discussion boards, asynchronous discussion 






Tania JAROSEWICH, 
Censeo Group, USA 
Lori VARGO 
The University of Akron, USA 
James SALZMAN 
Ohio University, USA 


Lisa LENHART, LeAnn KROSNICK 

The University of Akron, USA 
Kristen VANCE 
Cleveland State University, USA 
Kathleen ROSKOS 
John Carroll University, USA 


MS5W - 

W9im ■■ • mm 

•• (Puntambeker) 

(Bloom) 

= ibs ' ssp - mm - - fwiniiiia • 

w&mm ■■ mum 

- -naffia ’ ° 


1*7: ' wife®' immm 


118 


















Say What? The Quality of Discussion Board Postings in Online Professional Development 


Introduction 

Some of the most gratifying moments that we as 
learners have in classrooms are those times when a 
discussion prompts spirited and insightful exchanges 
that cause us to think and question our own beliefs. 
We feel energized, and the hum of the conversations 
lingers beyond the classroom. Sometimes, the hum 
is literal, as we converse while pouring into the 
hallways. Sometimes more vicariously, offering 
opportunities to relive high points and think of how 
we wished we’d responded. While the back-and- 
forth of the dialectic requires a quick mind and 
assertiveness to jump into the conversation, the 
world of the online classroom threatens many of us 
because it alters the traditional interaction of face- 
to-face, providing a different context that challenges 
us to retain the vigor of personal interaction in a 
virtual classroom. This same challenge is evident 
with the increasing use of on-line formats for teacher 
professional development. 

Although the technology for robust online teacher 
professional development (OTPD) is at our fingertips 
(and ubiquitous), far less is known about how to 
use it to advance teachers’ thinking, reasoning and 
instructional skill through professional development 
(McCombs & Vakili, 2005). Barab, Kling, and Gray 
(2004), for example, cite how little we know about how 
to build online learning communities that both inform 
teachers, and also help them solve practical problems 
of instruction. Further, different configurations of 
OTPD including elements such as length of training, 
instructional methods used in the training, and blended 
approaches that include both face-to-face and online 
components can have different impacts on learners 
(Bernard et al, 2004; Sitzmann, Kraiger, Stewart, and 
Wisher, 2006). A recent meta-analysis of web-supported 
learning studies showed that online learning is modestly 


more effective than traditional face-to-face instruction 
(US Department of Education, 2009). However, the 
authors of that study cautioned that simply offering an 
existing course in an online setting will not increase student 
achievement, but that the course should be redesigned to 
allow for additional learning opportunities, in particular, 
opportunities for self-reflection and self-assessment. This 
result is consistent with the cross-disciplinary research on 
human learning, the How People Learn (HPL) framework 
(Bransford, Brown & Cocking, 2000), which proposes 
three learning principles: (a) prior understandings influence 
new learning; (b) new understanding requires development 
of factual knowledge and conceptual frameworks; and 
(c) self-monitoring and reflection strategies support 
learning with understanding. Applying these principles 
to instruction calls for a learning environment that is 
learner-centered, building on what learners already 
know; knowledge-centered, emphasizing authentic 
achievement and mastery (Newmann & Associates, 
1996); assessment-centered, offering multiple means 
for monitoring learning progress; and community- 
centered, encouraging social networks and 
collaborative teams that support learning (Schlager 
& Fusco, 2004). 

Previous Research on Discussion Boards 

Asynchronous discussion boards provide an 
opportunity for participants in online courses to 
engage with course content and extend their learning 
through the process of discussion (Blumfield et al., 
1996). In an online discussion board, participants 
can reflect on previous postings and develop a 
thoughtful response or analysis of course content in 
an environment that can be less intimidating than 
face-to-face classroom discussions (Alvarez-Torres, 
2001; Meyer, 1996). Also, while discussions in 
classrooms are dominated by instructor contributions, 
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online discussions allow greater student control and 
contribution (Harasim, 1987). Research has shown 
that deep engagement in online discussions can 
improve overall class perfonnance (Ellis et al, 2006; 
Ellis et al., 2008; Krentler & Willis-Flurry, 2005, 
Offir, Lev & Bezalel, 2008; Schellens & Valcke, 
2006; Schrier, 2004) and can help develop deep 
learning and critical thinking skills (Mauriano, 2006). 

What constitutes deep engagement? Simply 
requiring instructors to offer and students to 
engage in an online discussion does not guarantee 
high-quality discussion posts and interactions 
(Buraphadeja & Dawson, 2008; Dennen, 2005; 
McLoughlin & Mynard, 2009). If tasks, prompts, and 
instructor feedback are not properly structured, and 
students do not engage in higher-order thinking, then 
the potential for this useful tool will not be realized. 
This article presents the results of a research study 
to understand the extent to which online discussion 
boards supported and extended participant learning 
in a set of online professional development modules 
for teachers of reading and allowed the participants to 
engage more deeply with the course materials. 


Content and Delivery System of eRead Ohio 
Online Professional Development Courses 

The goal of e-Read Ohio is to create an online 
professional development system for teachers 
of preschool through high school students. This 
initiative, which began in 2003, was originally 
funded by the No Child Left Behind Act through 
a Reading First grant and led by The Reading 
First-Ohio Center for Professional Development 
and Technical Assistance in Effective Reading 
Instruction. e-Read Ohio represents a model unique 
in the nation for investing Reading First dollars in 
an in-state program, and expanding the opportunities 
across the state. Through funding from the Ohio 
Department of Education, e-Read Ohio has expanded 
to provide effective and efficient high-quality online 
professional development in the area of literacy for 
Ohio’s teachers, select and train online instructors 
across the state, and build Ohio’s learning objects 
repository for professional development. 

Table 1 shows the stages of the project from 
developing and studying the first online module 
through the current year’s implementation schedule. 


Table 1. 

Timeline of eRead Ohio Project 


School Year 

Activity 

Course 

N Schools 

N Participants 

2004-2005 

Developing and deploying first course 
Research study comparing 
implementation models: F2F, online, and 
blended 

Scaffolding for Indiv 
Instruction 

12 

816 

2005-2006 

Pilot implementation 

Scaffolding for Indiv 
Instruction 

3 

30 

2006-2007 

Pilot implementation 

Scaffolding for Indiv 
Instruction 

Five Essential 
Components of 
Reading Instruction 

12 and 100 
individual 
particip in 
online course 

406 

2007-2008 

Statewide Implementation 

All courses 

119 

2,305 

2008-2009 

Statewide Implementation 

All courses 

329 

8,230 

Total 



475 

11,787 
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Since 2004, eRead Ohio has offered fourteen 
different courses to 11,787 teachers in 475 school 
districts (see Table 2). Teachers from every school 
in Ohio can participate in the program by learning, 
practicing, and applying techniques with measurable 
literacy benefits for elementary and secondary 
students in their classrooms. The e-Read Ohio menu 
of online course modules covers the most important 
topics and techniques in reading instruction, including 
differentiating instruction (DI), scaffolding, and 
the five essentials of reading (phonemic awareness, 
phonics, vocabulary, fluency, and comprehension). 
Teachers in e-Read Ohio’s online modules learn 


interactively in simulations, video analyses, and case 
studies. They watch expert video, view classroom 
footage to observe model teaching in action, and 
apply literacy techniques from the modules into 
their class instruction. They work individually and 
in online discussion groups. Each course includes 
an online module, face-to-face sessions, and an 
asynchronous online discussion board. Participants in 
the online courses are required to engage in the online 
discussion by responding to instructor prompts and 
follow-up questions, responding to other participants’ 
comments, and posting their own questions. 


Table 2. 

eRead Ohio Course Offerings 


Course Title 

Module/s 

Sequence 

Contact 

Hours 

Graduate 

Credit 

Grade 

Levels 

Adolescent Literacy in a 
Changing World 

1. Adolescent Literacy in a 
Changing World 

Tech-F2F-Online- 
F2F-Online-F2F 

15 

1 credit 

6-12 

Assess-Plan-Teach in 

Action 

1. Assess-Plan-Teach 

Tech-F2F-Online- 

F2F-Online-F2F 

12 

1 credit 

K-6 

Building Comprehension 
in Grades 4-6 

1. Fluency 

2. Vocabulary, 

3. Comprehension 

Tech-F2F-Online- 
Online-F2F-Online- 

F2F 

16 

1 credit 

4-6 

Developing Adolescents’ 
Content Vocabulary and 
Word Power 

1. Developing 

Adolescents’ Content 
Vocabulary and Word 

Power 

Tech-F2F-Online- 
F2F-Online-F2F 

15 

1 credit 

6-12 

Differentiating Instruction 
for Diverse Student Needs 

1. Differentiating 

Instruction 

F2F-Online-F2F- 

Online-F2F-Online- 

F2F 

12 

1 credit 

K-8 

Early Language Learning 

1. Oral Language 

2. Phonemic Awareness 

3. Phonics 

Tech-F2F-Online- 
F2F-Online-Online- 

F2F 

15 

1 credit 

PreK-3 

Five Essential 

Components of Reading 
Instruction 

1. Phonemic Awareness 

2. Phonics 

3. Fluency 

4. Vocabulary 

5. Comprehension 

Tech-F2F-Online- 

Online-F2F-Online- 

Online-Online-F2F 

24 

2 credits 

K-6 
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Fostering Students’ 
Comprehension of Texts 

1. Fostering Students’ 
Comprehension of Texts 

Literacy Coaching in 
Schools 

1. Literacy Coaching in 
Schools 

Scaffolding for Individual 
Instruction 

1. Differentiating 
Instruction 

2. Scaffolding 

School Leadership: 
Leadership for Learning 

1. Leadership for Learning 

School Leadership: 
Professional Learning 
Teams 

1. Professional Learning 
Teams 

Supporting the Adolescent 
Reader as Strategist 

1. Supporting the 
Adolescent Reader as 
Strategist 

Supporting English 
Learners in the Classroom 

1. English Learner 
Language 


Tech-F2F-Online- 

F2F-Online-F2F 

15 

1 credit 

6-12 

Tech-F2F-Online- 

F2F-Online-F2F 

15 

1 credit 

K-8 

Tech-F2F-Online-Sm 
Group Mtg-Online- 
F2F-Online-Sm 

Group Mtg-Online- 
F2F 

24 

2 credits 

K-5 

Tech - Online 

15 

1 

K-12 

Tech - Online 

15 

1 

K-12 

Tech-F2F-Online- 

F2F-Online-F2F 

15 

1 credit 

6-12 

Tech-F2F-Online- 

F2F-Online-F2F 

12 

1 credit 

K-8 


eRead Ohio’s evaluation and research agenda 
has examined several aspects of the online course 
delivery system. An early study found that student 
learning in blended and online courses was similar 
to that of course content delivered face-to-face 
(Roskos et ah, 2007b). The results of this study are 
similar to those of other studies that have found that 
online learning is as effective as or more effective 
than face-to-face instruction (US Department of 
Education, 2009) and supported the development of 
an online delivery system for statewide professional 
development. 

The eRead Ohio team’s analysis of course 
content showed that despite the goal of the course 
developers for a constructivist approach to learning, 
the online modules have a didactic pedagogy (show 
and tell) rather than a constructivist one (teaching for 
understanding, Roskos, et ah, 2007). The modules 
emphasize declarative types of knowledge (facts 
and concepts), and lower levels of cognitive demand 


(remember and understanding). Although the modules 
provide a limited level of learner participation, 
control, productivity, and creativity of experience, the 
sensory design of the modules was well developed 
and supportive of learner engagement. As a result of 
this research study, the team applied new instructional 
design methods during the development of a third, 
new module, to provide learners with more guided 
and self-initiated interaction. The analysis of the 
third module indicated that in fact, the design of the 
module was improved. This result suggested that 
modifying elements of online course design may 
provide a more constructivist environment. 

Although the third module showed stronger 
design elements that included assignments and 
activities that were more challenging and allowed 
for greater learning of content materials, the 
study suggested a need for stronger strategies to 
support constructivist principles of instruction and 
learning in the online environment. The eRead Ohio 
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team hypothesized that the modules present the 
basic course information and that the supporting 
materials, including the online discussions, provide 
opportunities for higher levels of cognitive challenge 
and interactive engagement in course content. 

Asynchronous Discussion Boards in the eRead 
Ohio Online Courses 

Each eRead Ohio module includes a discussion 
board whose goals are to increase collaboration and 
interaction among participants, increase knowledge 
acquisition, provide an opportunity for professional 
discussions, and create a statewide conversation on like 
topics to establish a common knowledge base. Through 
the online discussions, participants have the opportunity 
to analyze how their practice reflects the online 
course content and how to improve their instructional 
strategies. At the time of the project, e-Read Ohio 
project instructors received basic training in how to 
facilitate discussion boards and also received prompts 
that they could use in starting discussions. 

The goal of this study is to understand the extent to 
which the discussion boards in the eRead Ohio online 
professional development courses support and extend 
learning in the modules and allow students to engage 
more deeply with the course materials. The study 
investigated the following research questions: What are 
the characteristics of the discussion board prompts? To 
what extent do participants exhibit rigor and relevance 
in their online postings? Do postings show evidence of 
higher-order thinking? Across the online modules, what 
percentage of discussion is thoughtful or deeply engaged 
with the course content? What is the influence of the 
prompt on discussions? What do we observe about the 
role of the facilitator in the discussion? Do specific types 
of interjections lead to higher-level discussions? What 
kinds of facilitator interactions lead to thoughtfulness? 


Method 

Sample 

Researchers analyzed discussion boards from 
three of the 150 eRead Ohio online courses offered in 
the 2006-2007 school year. The three courses, taught 
by different instructors (Instructor A, Instructor 
B, and Instructor C), all covered the same content 
area, and were offered at approximately the same 
time in the school year. At the start of the study, the 
research team read the discussion board transcripts to 
familiarize themselves with the content and quality 
of the discussions. The team met to discuss initial 
impressions of the discussions and decide which 
instructors’ discussions to analyze. At this initial 
meeting, the team members ranked the quality of 
the discussions from high to low based on their 
initial impressions of the quality of the discussions. 
The research team, also the development team of the 
online course system, was familiar with the quality of 
the instructors. Each of the instructors was chosen as 
an e-Read Ohio facilitator because they had successful 
track records of professional development deployment. 
Instructors did receive instruction in the area of 
developing discussion online. The instructors were 
chosen to represent a range of skill in facilitating 
online discussions with an instructor considered to be 
strong, average, and low chosen for this study. 

Instrument 

Various content analysis rubrics and frameworks 
have been used for coding online discussions (see 
for example DeWever et al, 2006, and Meyer, 2006). 
Early research on discussion board participation 
simply examined quantitative data about levels 
of participation while later research went beyond 
level of participation to examine the content of the 
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discussions, social construction of knowledge, and critical 
thinking (DeWever et al., 2006). Few, if any studies used 
common instruments and therefore, results were neither 

Table 3. 

Categories for Coding Discussion Board Postings 


replicated nor extended. This lack of replication 
has been identified as one of the deficits in the area of 
analysis of discussion boards (Vlake & Martens, 2006). 


Code Description 

0 Contributions are shallow, reflecting little or no thought about the topic of discussion 

2 Contributions reflect limited thoughtfulness, comments are primarily opinions that are not supported by 
appropriate theory or course material 

^ Contributions show thoughtfulness; comments are primarily opinions, however there is limited support from 
theory and course materials 

6 Contributions show thoughtfulness; opinions are supported by appropriate theory and course materials 


Contributions are extremely thoughtful; opinions are supported with appropriate course material. Other 
8 participant comments along with personal experiences are synthesized and incorporated into a well- 
developed argument. 


In this study, we considered several existing 
instruments in order to build on previous work in 
this area of study. Among the instruments that the 
team reviewed were Puntambekar’s system (2006), 
the SOLO Taxonomy (Biggs & Collins, 1982), and 
the Garrson’s Four Cognitive Processing Categories, 
all of which have been used to categorize content of 
discussion boards (Mauriano, 2006; Meyer, 2006). 
The team ultimately chose to use two different 
classification systems. The first was the coding 
system developed by Puntambekar (2006, see Table 
3), which allows for coding of cognitive demand 
and also the extent to which the posting is related to 
theory or course material. The second classification 
system used in this study was the updated Bloom’s 
taxonomy (Anderson & Krathwohl, 2001), which 
specifies six levels of cognitive demand: remember, 
understand, apply, analyze, evaluate, and create. 


Procedure 

The text of each discussion board conversation 
was formatted to allow for consistent coding. 
The initial instructor prompt was listed first, with 
each response to the prompt listed below in the 
order in which it was posted. Discussions on non- 
academic topics such as the first posting in which 
students introduced themselves, and non-threaded 
discussions in which students reported on data that 
they collected in their classroom for a particular task 
were eliminated. Each of the remaining prompts 
was numbered for each instructor. Using an online 
random number generator, the researchers chose six 
numbers and selected the prompts in each course that 
corresponded with the random numbers. These six 
discussions in each online course were analyzed. 

Two members of the research team independently 
assigned a rating to each posting to reflect the 
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highest coding category evident in the posting using 
Puntambekar’s system and Bloom’s taxonomy. The 
unit of analysis was an entire posting (De Wever, 
Schellens, Valcke, & Van Keer, 2006). After the 
independent coding process, the two-person team 
compared coding results. In instances where the 
ratings were not the same, the members of the team 
came to consensus for a final rating. The inter-rater 
agreement between the teams was 72%. 

Results 

The results are reported in three separate 
sections. First, the authors discuss the characteristics 
of the discussion prompts. Second, they analyze the 
rigor of the student postings. Finally, they analyze the 
rigor of the instructor postings. 

Characteristics of Discussion Prompts 

The three discussion boards included in the study 

Table 4. 

Characteristics of Discussion Board Questions 


were comprised of 1,273 entries by 88 participants 
in response to 31 instructor questions. The average 
level of instructor prompts ranged in level on the 
Puntambekar rubric from 4.19 to 4.87 (See Table 4). 
That is, according to the coding rubric, the instructors 
asked students to provide thoughtful comments but in 
general, did not specifically ask students to use theory 
and course material to support their statements. 
The level of each prompt on the Bloom’s revised 
taxonomy was also coded. The prompts ranged from 
asking students to apply (Bloom’s level 3) to evaluate 
(Bloom’s level 5) the course content. Significant 
differences were noted in the level of questions that 
each of the three instructors asked (F (2, 1303) = 
52.135, p=. 000, rf=07). Post hoc tests revealed that 
the questions posted by Instructor B were at a lower 
level on the Puntambekar rubric than those posted 
by Instructor A (/?=.000) and Instructor C (p=. 000) 
although the effect size for this difference was small. 



Instructor A 

Instructor B 

Instructor C 

Total 

N questions 

11.00 

12.00 

8.00 

31.00 

Average rubric level 

4.88 

4.19 

4.87 

4.54 

Average Bloom’s level 

4.70 

3.48 

4.74 

4.13 


The average code assigned using the Bloom’s 
revised taxonomy was also different among the 
three instructors (F (2, 1303)=144.246, p=. 000, rf 
=.18) with a relatively small effect size. Post hoc 
tests revealed that, consistent with the coding using 
Puntambekar’s system, the questions posted by 
Instructor B were at a lower level on the Bloom’s 


taxonomy than those posted by Instructor A (p= .000) 
and Instructor C (p=.000). 

The following are two examples of question 
prompts posted by instructors: 

One purpose of professional development 
courses such as this is to bring about “reflection” 
upon and “growth” in classroom practice, 
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leading to gains in student achievement. From 
this module, identify one aspect that caused you 
to stop and reflect upon your own instruction. 
As a result of your reflection, what change(s) 
will you incorporate into your work? How will 
that benefit your students? 

This time, think in terms of content area 
reading. 1. Consider the “Close-Up Strategies” 
of context clues, structural analysis, and 
graphic organizers that are modeled in the 
videos. 2. Explain why one of these strategies 
would be essential for students to use as they 
read in science, social studies, math, or other 
content areas 

Rigor of Student Postings 

Student and instructor contributions to the online 
discussion were identified as either a new comment 
or a response to a previous comment. New comments 
were postings that responded to the prompt but 
did not build on or respond to a previous posting. 
Responses to previous comments were identified as 
those that referenced another posting. The overall 
average rating for student postings was 2.70 on the 
8-point coding scale (see Table 5), with the average 
code for new comments 3.56 and for postings that 
responded to previous comments 1.49. 

in several cases, postings that were offered in 
response to previous comments were thoughtful. For 
example, students offered each other resources and 
advice, as did this student, 

“Nora, 1 have a book that does give you the 
scope and sequence of phonics instruction. It 
progresses from consonant sounds to vowel 
sounds (short then long). Then they move on 
to vowel digraphs and diphthongs, consonant 
digraphs, consonant blends, onsets and rimes. I 


will bring it to class and we can take a look 
at it.” 

Other responses extended a previous comment. 
For example, 

“Leora, I do agree with you that automaticity 
aids comprehension. From my observation, 
those students who have attained automaticity 
in decoding words, have a longer focus in 
reading. On the other hand, the students who 
are still struggling give up easily. Most often, 
the struggling students spend such a long time 
decoding words that they forget what they had 
read before.” 

However, the vast majority of the postings in 
response to another comment were to encourage or 
support without addressing the content of the course 
or furthering the concept. The following examples are 
similar to many of these types of posts, 

“Stella, Excellent activity. I’m glad to see 
that you have found ways to use phonemic 
awareness with your 4 th graders. Good luck!” 
and 

“Robin, 1 like the idea of using the word in a 
sentence. It helps the student own the word.” 
The greatest percentage of student comments, 
42%, was coded at Level 4 of the rubric. This average 
rating reflected comments that were primarily 
descriptions of instructional practices with no support 
of theory or reference back to course material. 
Thirty-three percent of postings were rated at Level 
2 - generally a statement referring to and agreeing 
with colleagues’ comments - and 19% at Level 0 - 
unrelated or only tangentially related to the question. 
Of the student postings, 6% were rated at the two 
highest levels of the rubric. These 68 statements were 
made by 32 of the 88 students. 
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Table 5. 

Average Ratings of Student and Instructor Comments 



Instructor 1 

Instructor 2 

Instructor 3 

All postings 


Student postings 



New comment 

4.30 

2.89 

3.73 

3.56 


(N=164) 

(N=256) 

(N=285) 

(N=705) 

Response to previous comment 

2.00 

1.27 

1.74 

1.49 


(N=53) 

(N=342) 

(N=77) 

(N=473) 

All student comments 

3.74 

1.96 

3.30 

2.70 


(N=217) 

(N=599) 

(N=362) 

(N=l,178) 


Instructor Postings 



New comment 

2.00 

2.67 

4.00 

3.40 


(N=2) 

(N=6) 

(N= 12) 

(N=20) 

Response to previous comment 

2.17 

1.85 

3.53 

3.00 


(N=12) 

(N=13) 

(N=47) 

(N=72) 

All instructor comments 

2.14 

2.20 

3.67 

3.14 


(N=14) 

(N=20) 

(N=61) 

(N=95) 


A one-way ANOVA indicated significant 
differences among the participant conversations in 
the three instructors’ courses (F (2, 1177)= 148.88, 
p=. 000, if=.20), with a moderate effect size. The 
Tukey post hoc analysis indicated that the student 
postings in the course led by Instructor A were 
significantly higher (p=.000) than those of Instructor B 
(p=.000) or Instmctor C (p=.000) and those of Instructor 
C were higher than those of Instructor B (p=.000). A 
significant correlation was observed between the coded 
level of the prompt and the average coded student 
posting for the prompt (r=,198, j9=.000) indicating 
that when a prompt was at a higher level of the rubric, 
students provided responses at higher levels. 

Rigor of Instructor Postings 

The comments that instructors posted were 
also coded on the 8-point Puntambekar rubric. The 
average code for the instructors’ comments was 3.14, 
with a range of 2.14 to 3.67 (see Table 5). A one¬ 
way ANOVA indicated significant differences among 


the three sections of the course (F (2, 94)=11.447, 
p=. 000, rf=.20), with a moderate effect size. The 
Tukey post hoc analysis indicated that Instructor C 
posted comments that were at a higher level on the 
rubric than did Instructor A (p=. 002) or Instructor 
B (f=.001). The frequency with which Instructor C 
engaged in the discussion board was higher than that 
of the other two instructors. 

Similar to the student results, the greatest 
percentage of instructors’ comments, 54%, was coded 
at Level 4 of the rubric. This average rating reflected 
comments that were primarily related to instructional 
practices with no support of theory or reference back 
to course material. Twenty-five percent of postings 
were rated at Level 2 and 13% at Level 0. Of the 
instructor postings, 7% were rated at the two highest 
levels of the rubric. 

Examples of instructor posts are included below: 

Kelly, I just love your lesson!!! Way to get the 

children engaged. 

Nora has posed several intriguing questions 
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to the class... any insights? 1.) How are others 
using word study in their classrooms? 2.) What 
assessments are you using to determine the 
direction to go? 

Judy, 1 understand your frustration with helping 
students make the leap from making sounds to 
demonstrating it on paper. Any ideas from the 
esteemed group? 

Discussion 

Several factors appear to have an influence on 
the quality of online discussions. Two factors appear 
to account for a good deal of the variance. First, 
the characteristics of the discussion prompt set the 
parameters of the exchanges that follow. Second, it 
is clear that merely participating is not sufficient to 
ensure the level of rigor that course instructors expect 
of graduate students and practicing teachers. In this 
section, the authors discuss these two areas, discuss 
the implications of the Puntambekar coding system, 
and offer recommendations for further research. 

Characteristics of Discussion Prompts 

In all of the discussion boards that were coded, 
the most significant finding was that although 
differences were evident among the level of prompts 
posed by the instructors, the prompts themselves and 
the responses to those prompts did not reach high 
levels of the coding rubric. The majority of prompts 
requested participants to reflect on classroom use of 
instructional practices described in the course, but 
did not explicitly instruct participants to connect 
these practices back to the course content or relevant 
theory. Although instructors were given a set of 
sample prompts that they could use, in the sample 
of discussion boards analyzed in this study, the 


instructors did not use the prompts offered by eRead 
Ohio, instead each instructor developed her own 
prompts to begin the discussions. Also, the instructors 
often used the same type of prompt for different 
discussion topics in the course. For example, one 
instructor used a prompt that asked the participants to 
select a student or a small group of students, explain 
(in one or two bullet points) why they selected these 
students, describe (in one or two bullet points) the 
data used to assess their skills (phonemic awareness, 
phonics, fluency, etc.), plan and implement an activity 
based upon these data, and briefly outline the activity. 

The level of the question that instructors asked 
was correlated with the level of responses that 
students produced. Questions that were rated at higher 
levels on the scoring rubric, in general generated 
responses that fell at higher levels on the rubric. 

Rigor in Postings 

The discussion boards were relatively lively in 
terms of number of postings and interactions between 
participants. Students reported increased insight into 
their practice and many described intended changes 
in practices. However, the vast majority of discussion 
prompts did not insist that students reflect and comment 
on course content in their response and few participants 
referred to course content to support their current 
practices or as a basis for changing their practice. 

In addition to the lack of challenge in the 
discussion prompts to engage with the course materials, 
the majority of instructor comments on the discussion 
boards did not challenge students to provide a more 
comprehensive or thoughtful answer. In one case in 
which the instructor asked follow-up questions that 
challenged the group to consider a topic in greater detail, 
several participants posted a response to the original 
question, presumably to complete the assignment, but 
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nobody posted responses to the follow-up questions. 
In this sample of discussion boards, the initial 
question had the greatest impact on participants’ 
thoughtfulness, with little impact of instructors’ 
follow-up questions or postings. 

The Need for Modeling by Instructors 

In general, the instructors did not model for 
participants how to create a high-level posting. The 
majority of the instructor responses encouraged students 
to continue the discussion at the current level, even 
though the responses were not as connected to the 
course content as they could have been. Perhaps the 
instructors themselves were unaware of the architecture 
of a strong online discussion board post or perhaps 
they felt unsure about how to challenge deeper 
thinking about the content on an online forum. 

Although participants were instructed to 
respond to one another, they were not given specific 
information about the types of interactions that were 
expected. Postings that were an initial response to the 
prompt received higher ratings than did postings in 
response to previous comments. Participants did not 
often refer to each other’s postings when developing 
their own response to a prompt except when to praise 
or congratulate a colleague for a lesson that they 
thought successful or interesting. 

The primary way in which students learned from 
one another on the discussion boards appeared to be 
in gaining ideas about new or different instructional 
practices that they might wish to add to their own 
repertoire. Students did not spontaneously apply the 
practices of reflection on instructional theories or 
challenging assumptions of other participants. The 
results of this study suggest that if one of the goals 
of a discussion board is that participants engage 
with the course content at higher cognitive levels 


and challenge each other’s thinking, that discussion 
prompts must better facilitate those behaviors. 

Contextual Factors Implicated in Results 

Several aspects of the course delivery and 
organization may have led to the observed results. 
First, the participant groups were teachers who 
taught at the same school, and therefore, had many 
opportunities for professional discussions during the 
work day. In the online posts, a number of participants 
referred to discussions that they had about the course 
content in face-to-face discussions. Data collected 
from other stakeholders, including principals in 
schools in which courses were offered, indicated 
that participants were engaging with the course 
content during discussions and interactions with 
their peers during the school day, and postings on the 
discussion board were simply to fulfill a requirement. 
Also, the courses were offered with pass/fail credit, 
not with assigned grades. Therefore, participants 
might have felt lower levels of accountability with 
regards to the level of postings and interaction on 
the online discussion boards. The third factor that 
may have affected participation and engagement in 
the discussion boards was that participants did not 
necessarily come to the course voluntarily, it could 
be that a principal selected the course for the group of 
teachers and not every teacher may have agreed that 
this was a valuable experience. 

One more point to note is that during the time 
period from which these data were collected, participants 
were not informed of the rubric that would be used 
to analyze their postings. Since the time of this study, 
the eRead team has provided more in-depth training 
for instructors on how to facilitate and manage 
discussion board activities including the expectations 
that instructors were to interact frequently in the 
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discussion boards, post timely responses to queries, 
scaffold meaningful discussions, develop high- 
quality discussion board prompts and responses, 
and develop rapport on the discussion board with 
participants. The team developed a rubric (see Table 


6), which is available to instructors and students to 
inform all participants of expectations with regards to 
engagement in the online discussion boards. 

Table 6. 

Rubric for High Quality Online Posting from eRead 


Ohio Five Essential Components of Reading Instruction Course 



4 

3 

2 

i 

Critical thinking 

• Rich in content 

• Full of thought, insight, analysis 

• Includes probing question or 
comment/question to extend 
others’ thinking, leading to 
thoughtful responses 

• Substantial information 

• Evidence that thought, 
analysis has taken place 

• Responds to others’ 
postings in thoughtful 
manner 

• Generally competent 

• Information is thin, 
commonplace, 
general knowledge 

• Rudimentary and 
superficial 

• No analysis or 
insight displayed 

Connections / 
storytelling 

• New ideas or new connections: 
-To previous or current 
situations 

-To real-life situations 

• Rich in detail 

• Clear connections to 
previous or current 
situations 

• Lacks depth or detail 

• Connections limited, 
if any, 

• Vague generalities 
(“safe” comments) 

• Connections not 

made 

• Off topic 

Uniqueness / 
rumination 

• New ideas, applications 

• New connection 

• Unique personal opinion 

• Made and developed with depth 
and detail 

• References and/or resources to 
support all comments 

• New ideas, applications, 
or connections 

• Personal opinion 

• Lack depth and/or detail 

• Some references and/or 
resources mentioned 

• Few if any new ideas 
or connections 

• Rehash or summary 
of other postings 

• No new ideas 

• “I agree with...” 

statement 

Timeliness / 
participation 

• All required postings 

• Early in discussion 

• Throughout discussion 

• All required postings 

• Some not in time for 
others to respond 

• All required postings 

• Most at last minute, 
not allowing for 
responses from 
others 

• Some, or all, 
required postings 
missing 

References / 

resources 

• Appropriately cited relevant 
ideas beyond the assigned 
readings 

• Appropriately referenced 
class lectures, notes, 
material or readings 

• No citations or 

references 

• Inaccurate 

citation or 
misapplied 
reference 


Implications of the Puntambekar System for Coding Online Discussions 


The analysis of postings to the online professional 
development discussion boards provided the course 
development team with information about the quantity 
and quality of discussions that were occurring in the 
courses. The level of discussions among students - 
focused on instructional practices but not connecting 
those practices to course theory - was similar to the 
pattern found by Puntambekar (2006), whose coding 


system was used in the current study. Similar to 
Puntambekar’s findings, the participants in the current 
online course answered the initial question posed 
by the instructor and did not challenge each other 
to higher levels of analysis or reflection. This study 
adds to the research in the area of analysis of online 
discussions by connecting instructor postings with 
participant engagement in the discussion board, using 
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a previously-developed coding system. The results of 
the study can be applied to training for online course 
instructors and also for course development. 

On a larger scale, the methods applied for 
analysis of online discussions in this study provide 
an example of how evaluators and educators can use 
a clear coding system to quickly analyze discussion 
board activity during a course or after the completion 
of a course, how instructors can analyze discussion 
boards as they are facilitating a course, and how 
participants can monitor themselves. The question 
remains of how instructors or developers of online 
courses can leverage teachers’ desire to talk about 
their work to discussions in an online setting. What 
types of prompts could help teachers connect their 
practice to theory and examine their practice against 
the backdrop of theory within the structure of an 
online discussion forum. 

Recommendations for Future Research 

Just as with face-to-face discussions, teachers and 
program developers should probably not be surprised 
by the results. That is, if instructors want students to 
get to higher levels of interaction with the course 
content, they need to model the behaviors expected 
and explicitly ask them to connect course content to 
practice. This study became a good reminder to us that, 
just because course content and delivery enters a new 
context, it does not mean that the fundamentals of 
good instruction do not need to be modeled within 
the vicarious world of online learning. 

To that end, course developers and instructors 
of online courses are provided the following 
recommendations based on the findings of this study: 

• Developers should provide instructors with 
more explicit directions about effective 
prompts and give a formula to instructors to 


follow when writing a prompt. 

• Developers should provide models for 
instructors of how to challenge students to 
higher levels of thinking and analysis of 
course content. 

• Professional development for instructors 
should address the expectations regarding 
questions, student engagement, and 
instructor behavior on the discussion board, 
as well as information about level and types 
of conversations and the types of responses 
to previous messages that are expected. 

• Course providers should monitor instructors’ 
success at generating high-level questions and 
encouraging high-level student responses. 

• The study did not examine student characteristics 
with regard to the study’s findings. It is likely 
that the time that students have to devote, 
reasons for taking a course, and students’ drive 
will affect the quality of the posting and these 
motivational and personal characteristics might 
be considered in subsequent studies. 

• Facilitator’s course load could be a factor to 
consider with regard to their effectiveness in 
engaging in discussion boards. Instructors 
who are teaching concurrently many different 
courses with high numbers of participants 
might perform differently from those with just 
one course or courses with few participants. 

• Instructors’ knowledge of course content or 
familiarity with the course could influence 
level of questions. Therefore, instructors must 
be trained and provided support during the 
time that they are facilitating a course to ensure 
that they have the capacity to support learners’ 
engagement in the discussion boards. 
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